Filed by: csabou [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=377223543&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/X0iTjsUWR-if9bQWu_Kywg/runs/0/artifacts/public/logs/live_backing.log

[task 2022-05-07T04:39:16.348Z] 04:39:16     INFO -  check>           ld.lld: error: undefined symbol: Gecko_Destroy_nsStyleDisplay
[task 2022-05-07T04:39:16.348Z] 04:39:16     INFO -  check>           >>> referenced by gecko_properties.rs:17418 (/builds/worker/workspace/obj-build/x86_64-unknown-linux-gnu/release/build/style-185d00fd7120ca78/out/gecko_properties.rs:17418)
[task 2022-05-07T04:39:16.349Z] 04:39:16     INFO -  check>           >>>               style-c0031f2be142f9f5.style.22b6963e-cgu.0.rcgu.o:(core::ptr::drop_in_place$LT$servo_arc..ArcInner$LT$style..gecko_bindings..structs..root..mozilla..GeckoDisplay$GT$$GT$::h8f80291c2c49aea4) in archive /builds/worker/workspace/obj-build/x86_64-unknown-linux-gnu/release/deps/libstyle-c0031f2be142f9f5.rlib
[task 2022-05-07T04:39:16.349Z] 04:39:16     INFO -  check>           >>> referenced by gecko_properties.rs:17418 (/builds/worker/workspace/obj-build/x86_64-unknown-linux-gnu/release/build/style-185d00fd7120ca78/out/gecko_properties.rs:17418)
[task 2022-05-07T04:39:16.349Z] 04:39:16     INFO -  check>           >>>               style-c0031f2be142f9f5.style.22b6963e-cgu.0.rcgu.o:(core::ptr::drop_in_place$LT$alloc..boxed..Box$LT$servo_arc..ArcInner$LT$style..gecko_bindings..structs..root..mozilla..GeckoDisplay$GT$$GT$$GT$::h5e045007d7efc75e) in archive /builds/worker/workspace/obj-build/x86_64-unknown-linux-gnu/release/deps/libstyle-c0031f2be142f9f5.rlib
[task 2022-05-07T04:39:16.350Z] 04:39:16     INFO -  check>           >>> referenced by gecko_properties.rs:17418 (/builds/worker/workspace/obj-build/x86_64-unknown-linux-gnu/release/build/style-185d00fd7120ca78/out/gecko_properties.rs:17418)
[task 2022-05-07T04:39:16.350Z] 04:39:16     INFO -  check>           >>>               style-c0031f2be142f9f5.style.22b6963e-cgu.0.rcgu.o:(core::ptr::drop_in_place$LT$style..gecko_bindings..structs..root..mozilla..GeckoDisplay$GT$::hdad6ff1e0fe7dba3) in archive /builds/worker/workspace/obj-build/x86_64-unknown-linux-gnu/release/deps/libstyle-c0031f2be142f9f5.rlib
[task 2022-05-07T04:39:16.350Z] 04:39:16     INFO -  check>           >>> referenced 2 more times
[task 2022-05-07T04:39:16.350Z] 04:39:16     INFO -  check>
[task 2022-05-07T04:39:16.351Z] 04:39:16     INFO -  check>           ld.lld: error: undefined symbol: Gecko_Destroy_nsStyleEffects
[task 2022-05-07T04:39:16.351Z] 04:39:16     INFO -  check>           >>> referenced by gecko_properties.rs:17565 (/builds/worker/workspace/obj-build/x86_64-unknown-linux-gnu/release/build/style-185d00fd7120ca78/out/gecko_properties.rs:17565)
[task 2022-05-07T04:39:16.351Z] 04:39:16     INFO -  check>           >>>               style-c0031f2be142f9f5.style.22b6963e-cgu.0.rcgu.o:(core::ptr::drop_in_place$LT$servo_arc..ArcInner$LT$style..gecko_bindings..structs..root..mozilla..GeckoEffects$GT$$GT$::h1e0ff434227c9bd9) in archive /builds/worker/workspace/obj-build/x86_64-unknown-linux-gnu/release/deps/libstyle-c0031f2be142f9f5.rlib
[task 2022-05-07T04:39:16.352Z] 04:39:16     INFO -  check>           >>> referenced by gecko_properties.rs:17565 (/builds/worker/workspace/obj-build/x86_64-unknown-linux-gnu/release/build/style-185d00fd7120ca78/out/gecko_properties.rs:17565)
[task 2022-05-07T04:39:16.352Z] 04:39:16     INFO -  check>           >>>               style-c0031f2be142f9f5.style.22b6963e-cgu.0.rcgu.o:(core::ptr::drop_in_place$LT$alloc..boxed..Box$LT$servo_arc..ArcInner$LT$style..gecko_bindings..structs..root..mozilla..GeckoEffects$GT$$GT$$GT$::h55b5e1fbdc158638) in archive /builds/worker/workspace/obj-build/x86_64-unknown-linux-gnu/release/deps/libstyle-c0031f2be142f9f5.rlib
[task 2022-05-07T04:39:16.353Z] 04:39:16     INFO -  check>           >>> referenced by gecko_properties.rs:17565 (/builds/worker/workspace/obj-build/x86_64-unknown-linux-gnu/release/build/style-185d00fd7120ca78/out/gecko_properties.rs:17565)
[task 2022-05-07T04:39:16.353Z] 04:39:16     INFO -  check>           >>>               style-c0031f2be142f9f5.style.22b6963e-cgu.0.rcgu.o:(core::ptr::drop_in_place$LT$style..gecko_bindings..structs..root..mozilla..GeckoEffects$GT$::hfa44b6d20f82a650) in archive /builds/worker/workspace/obj-build/x86_64-unknown-linux-gnu/release/deps/libstyle-c0031f2be142f9f5.rlib
[task 2022-05-07T04:39:16.353Z] 04:39:16     INFO -  check>           >>> referenced 2 more times
[task 2022-05-07T04:39:16.353Z] 04:39:16     INFO -  check>
[task 2022-05-07T04:39:16.353Z] 04:39:16     INFO -  check>           ld.lld: error: too many errors emitted, stopping now (use -error-limit=0 to see all errors)
[task 2022-05-07T04:39:16.354Z] 04:39:16     INFO -  check>           clang-14: error: linker command failed with exit code 1 (use -v to see invocation)
[task 2022-05-07T04:39:16.354Z] 04:39:16     INFO -  check>
[task 2022-05-07T04:39:16.354Z] 04:39:16     INFO -  check>
[task 2022-05-07T04:39:16.499Z] 04:39:16     INFO -  check> error: could not compile `stylo_tests` due to previous error
[task 2022-05-07T04:39:16.499Z] 04:39:16     INFO -  check>
[task 2022-05-07T04:39:16.499Z] 04:39:16     INFO -  check> Caused by:
[task 2022-05-07T04:39:16.506Z] 04:39:16     INFO -  check>   process didn't exit successfully: `CARGO=/builds/worker/fetches/rustc/bin/cargo CARGO_CRATE_NAME=stylo_tests CARGO_MANIFEST_DIR=/builds/worker/checkouts/gecko/servo/ports/geckolib/tests CARGO_PKG_AUTHORS='The Servo Project Developers' CARGO_PKG_DESCRIPTION='' CARGO_PKG_HOMEPAGE='' CARGO_PKG_LICENSE=MPL-2.0 CARGO_PKG_LICENSE_FILE='' CARGO_PKG_NAME=stylo_tests CARGO_PKG_REPOSITORY='' CARGO_PKG_VERSION=0.0.1 CARGO_PKG_VERSION_MAJOR=0 CARGO_PKG_VERSION_MINOR=0 CARGO_PKG_VERSION_PATCH=1 CARGO_PKG_VERSION_PRE='' CARGO_PRIMARY_PACKAGE=1 LD_LIBRARY_PATH='/builds/worker/workspace/obj-build/release/deps:/builds/worker/fetches/rustc/lib' /builds/worker/fetches/rustc/bin/rustc --crate-name stylo_tests servo/ports/geckolib/tests/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --emit=dep-info,link -C opt-level=2 -C embed-bitcode=no --test -C metadata=a61104aee440ddbb -C extra-filename=-a61104aee440ddbb --out-dir /builds/worker/workspace/obj-build/x86_64-unknown-linux-gnu/release/deps --target x86_64-unknown-linux-gnu -C linker=/builds/worker/checkouts/gecko/build/cargo-linker -L dependency=/builds/worker/workspace/obj-build/x86_64-unknown-linux-gnu/release/deps -L dependency=/builds/worker/workspace/obj-build/release/deps --extern atomic_refcell=/builds/worker/workspace/obj-build/x86_64-unknown-linux-gnu/release/deps/libatomic_refcell-686d3eb5af9c4889.rlib --extern cssparser=/builds/worker/workspace/obj-build/x86_64-unknown-linux-gnu/release/deps/libcssparser-eabf96760c500b98.rlib --extern cstr=/builds/worker/workspace/obj-build/release/deps/libcstr-7add7f7ad9125fea.so --extern env_logger=/builds/worker/workspace/obj-build/x86_64-unknown-linux-gnu/release/deps/libenv_logger-2e73795cf4095161.rlib --extern euclid=/builds/worker/workspace/obj-build/x86_64-unknown-linux-gnu/release/deps/libeuclid-b22ca7090707a8a6.rlib --extern geckoservo=/builds/worker/workspace/obj-build/x86_64-unknown-linux-gnu/release/deps/libgeckoservo-0c5bde75c929fdb2.rlib --extern libc=/builds/worker/workspace/obj-build/x86_64-unknown-linux-gnu/release/deps/liblibc-463c0f44f07cfef5.rlib --extern log=/builds/worker/workspace/obj-build/x86_64-unknown-linux-gnu/release/deps/liblog-b923b2802b3d3072.rlib --extern malloc_size_of=/builds/worker/workspace/obj-build/x86_64-unknown-linux-gnu/release/deps/libmalloc_size_of-28dfc771c4f19ff6.rlib --extern num_traits=/builds/worker/workspace/obj-build/x86_64-unknown-linux-gnu/release/deps/libnum_traits-d77df0a748775366.rlib --extern selectors=/builds/worker/workspace/obj-build/x86_64-unknown-linux-gnu/release/deps/libselectors-03fde44f2ce82759.rlib --extern size_of_test=/builds/worker/workspace/obj-build/x86_64-unknown-linux-gnu/release/deps/libsize_of_test-8a7f6d2ccbf7fe4d.rlib --extern smallvec=/builds/worker/workspace/obj-build/x86_64-unknown-linux-gnu/release/deps/libsmallvec-0bf95395b780cd2d.rlib --extern style=/builds/worker/workspace/obj-build/x86_64-unknown-linux-gnu/release/deps/libstyle-c0031f2be142f9f5.rlib --extern style_traits=/builds/worker/workspace/obj-build/x86_64-unknown-linux-gnu/release/deps/libstyle_traits-08df1e7095e90e5f.rlib --extern to_shmem=/builds/worker/workspace/obj-build/x86_64-unknown-linux-gnu/release/deps/libto_shmem-89133b13ab156e99.rlib -C debuginfo=2 -C force-frame-pointers=yes --cap-lints warn -Ccodegen-units=1 -Zprofile -Cpanic=abort -Zpanic_abort_tests -Clink-dead-code -Coverflow-checks=off -C codegen-units=1` (exit status: 1)
[task 2022-05-07T04:39:16.511Z] 04:39:16    ERROR -  check> gmake[4]: *** [/builds/worker/checkouts/gecko/config/makefiles/rust.mk:469: force-cargo-test-run] Error 101
[task 2022-05-07T04:39:16.511Z] 04:39:16     INFO -  check> gmake[4]: Target 'check' not remade because of errors.
[task 2022-05-07T04:39:16.512Z] 04:39:16    ERROR -  check> gmake[3]: *** [/builds/worker/checkouts/gecko/config/recurse.mk:99: toolkit/library/rust/check] Error 2
[task 2022-05-07T04:39:16.512Z] 04:39:16     INFO -  check> gmake[3]: Target 'recurse_check' not remade because of errors.
[task 2022-05-07T04:39:16.512Z] 04:39:16    ERROR -  check> gmake[2]: *** [/builds/worker/checkouts/gecko/config/recurse.mk:34: check] Error 2
[task 2022-05-07T04:39:16.518Z] 04:39:16    ERROR -  gmake[1]: *** [/builds/worker/checkouts/gecko/build/moz-automation.mk:110: automation/check] Error 2
[task 2022-05-07T04:39:16.519Z] 04:39:16     INFO -  gmake: *** [client.mk:67: build] Error 2

style crate have several functions that reference C++ code, prefixed with Gecko_. objdump on style-*.rlib confirms that the symbols for these references indeed exist, unresolved. The crate is pulled in by stylo_tests, but since the tests never call these functions referencing to the C++ functions, they're considered dead and get dropped. Again, objdump of the stylo-tests executable show a lack of Gecko_* symbols.
The coverage build adds a flag -C link-dead-code, to keep a valid code coverage metric. Since we aren't linking the test binary with the C++ lib, the linkage fails.
Removing the flag from this code coverage target allows a successful run... but that's not ideal for Rust code coverage metric. Also, this flag was added a while ago as part of bug 1540749, and doesn't explain build successes prior to the failing build.

EDIT: Locally adding export RUSTFLAGS="-Clink-dead-code" cause rusttests to fail, with similar, but not identical, complaints about undefined references to the FFI functions. However, this is true even when the repo checkout is on changeset 4db3945f4fcba9a22aae42bdcabdf3525d46dc42 which is the last successful run for this target on try...

Seems like this used to work out of sheer luck, because nothing was directly or indirectly requiring those gecko symbols (there are subtle linking details about that). Rust tests cannot use gecko code. The gecko feature of the servo crate is not enabled when running these tests, so this works around the problem:

--- a/servo/ports/geckolib/tests/lib.rs
+++ b/servo/ports/geckolib/tests/lib.rs
@@ -32,4 +32,5 @@ extern crate euclid;
 #[cfg(target_pointer_width = "64")]
 mod size_of;
 mod specified_values;
+#[cfg(feature = "gecko")]
 mod piecewise_linear;

But that would essentially disable the piecewise-linear tests on our CI.

Generally speaking, the servo crate should probably have a gtest wrapper that runs the rust tests.

This should un-bust ccov builds for now. See comment for the follow-up
work.

Just to clarify, you mean something like the gtest setup the nsstring crate has, right? In practice I don't think we need that for the most part, see bug 1768874.

I proposed a stopgap solution for now, and will send patches to bug 1768874

https://hg.mozilla.org/mozilla-central/rev/d3359c870ff9

Just to clarify, you mean something like the gtest setup the nsstring crate has, right?

Kind of. I think we need a generic way to hook rust tests to gtests.